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Objective This study aimed to investigate whether the supplementation of Verbal Fluency: Animal category test (VF) performance 
can improve the screening ability of Mini-Mental State Examination (MMSE) for mild cognitive impairment (MCI), dementia and their 
major subtypes. 

Methods Six hundred fifty-five cognitively normal (CN), 366 MCI [282 amnestic MCI (aMCI); 84 non-amnestic MCI (naMCI)] and 
494 dementia [346 Alzheimer's disease (AD); and 148 non-Alzheimer's disease dementia (NAD)] individuals living in the community 
were included (all aged 50 years and older) in the study. 

Results The VF-supplemented MMSE (MMSE+VF) score had a significantly better screening ability for MCI, dementia and overall 
cognitive impairment (MCI plus dementia) than the MMSE raw score alone. MMSE+VF showed a significantly better ability than 
MMSE for both MCI subtypes, i.e., aMCI and naMCI. In the case of dementia subtypes, MMSE+VF was better than the MMSE alone 
for NAD screening, but not for AD screening. 

Conclusion The results support the usefulness of VF-supplementation to improve the screening performance of MMSE for MCI and 
NAD. Psychiatry Investig 2014;11:44-51 

Key Words Mini-Mental State Examination, Verbal fluency, Screening accuracy, Mild cognitive impairment, 
Non- Alzheimer's disease dementia. 



INTRODUCTION 

The Mini-Mental State Examination (MMSE) 1 is a widely 
used brief screening tool for the detection of cognitive impair- 



ment. In spite of its briefness and practical usefulness, it has a 
couple of important limitations for the early detection of cog- 
nitively impaired states. First, MMSE has poor sensitivity in 
detecting mild cognitive impairment (MCI), 2 ' 3 an at-risk state 
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for the development of dementia, 4,5 although early detection of 
the pre-dementia state is increasingly important, with high 
possibilities for the development of disease modifying treat- 
ments for Alzheimer's disease (AD) and related dementias. 
Second, MMSE consists of test items primarily covering orien- 
tation, attention, memory and language, and is less sensitive to 
frontal executive dysfunction. 2 ' 6 While memory decline is the 
earliest and most important cognitive deficit in amnestic MCI 
(aMCI) and AD, frontal executive dysfunction is frequently 
more prominent than memory or other cognitive deficits in 
non-amnestic MCI (naMCI) and non- Alzheimer's disease de- 
mentia (NAD), especially frontotemporal dementia (FTD) 7 
and vascular dementia (VD). 8 Therefore, the assessment of 
frontal executive dysfunction could make a meaningful contri- 
bution to the screening of overall MCI and dementia, or naM- 
CI and NAD. 

Verbal Fluency: Animal category test (VF) provides an as- 
sessment of semantic fluency by asking subjects to name as 
many animals as possible within 1 minute. 9 Semantic fluency 
has been shown to be reduced in subjects with MCI compared 
to those who are cognitively normal (CN). 10 Furthermore, se- 
mantic fluency involves not only the ability to search semantic 
memory using categorical rules, but also to look for the execu- 
tive function required to track prior responses and block in- 
trusions from other semantic categories. 11 These imply that 
VF-supplementation has been proposed to improve the MCI 
and dementia screening ability of MMSE. Nevertheless, the ef- 
fect of supplementing VF to MMSE score on the accuracy of 
MCI and dementia screening was poorly investigated. 

In this study, we aimed to investigate whether the supple- 
mentation of VF performance can improve the screening abil- 
ity of MMSE for MCI, dementia and their major subtypes. 

METHODS 
Subjects 

Study subjects were recruited from a pool of individuals 
registered in a program for the early detection and manage- 
ment of dementia at four centers located in Seoul, Korea (two 
public health centers, one senior citizens welfare center and 
one dementia clinic) from January 2000 to May 2011. In this 
study, 655 CN, 366 MCI (282 aMCI; 84 naMCI) and 494 de- 
mentia (346 AD; 148 NAD) individuals living in the commu- 
nity were included (all aged 50 years and older). 

A diagnosis of dementia was made according to the criteria 
of the fourth edition of the Diagnostic and Statistical Manual 
of Mental Disorders. 12 AD was diagnosed according to the 
probable or possible AD criteria of the National Institute of 
Neurological and Communication Disorders and Stroke/ Al- 
zheimer's Disease and Related Disorders Association 



(NINCDS-ADRDA). 13 VD was diagnosed according to the 
probable or possible VD criteria of the National Institute of 
Neurological Disorders and Stroke-Association Internationale 
pour la Recherche et l'Enseignement en Neurosciences 
(NINDS-AIREN). 14 Dementia with Lewy bodies (DLB) or 
Parkinson's disease dementia (PDD) was diagnosed according 
to the DLB consensus criteria, 15 and FTD was diagnosed ac- 
cording to the FTD consensus criteria. 16 MCI was diagnosed 
according to the current consensus criteria. 17 All MCI individ- 
uals had an overall clinical dementia rating scale (CDR) 18 of 
0.5. All NC subjects received a CDR score of 0. The exclusion 
criteria for all subjects included any present serious medical, 
psychiatric and neurologic disorders which could affect the 
mental function; the presence of severe behavioral or commu- 
nication problems which would make a clinical examination 
difficult; an absence of a reliable informant; and the inability of 
reading Korean [i.e., inability of reading 10 words in the Word 
List Memory from the Consortium to Establish a Registry for 
Alzheimer's disease (CERAD) neuropsychological battery 19,20 ]. 
All Individuals with minor physical abnormalities (e.g., diabe- 
tes with no serious complications, essential hypertension, mild 
hearing loss or others) were included. The Institutional Review 
Board of the Seoul National University Hospital, Korea ap- 
proved the study, and subjects or their legal representatives gave 
written informed consent. 

Clinical and neuropsychological assessments 

All subjects were examined by psychiatrists with advanced 
training in dementia research according to the CERAD proto- 
col. 19 ' 20 The CERAD clinical assessment battery included 
CDR, 18 Blessed Dementia Scale-Activities of Daily Living 
(BDS-ADL), general medical examination, neurologic exami- 
nation, laboratory tests, and brain MRI or computed tomogra- 
phy. The standard administration of the CERAD battery was 
previously described in detail. 19,20 Reliable informants were 
necessarily interviewed to acquire the accurate information 
regarding the cognitive, emotional and functional changes as 
well as the medical history of the subjects. 

MMSE, VF and other neuropsychological tests (15-item 
Boston Naming Test, Word List Memory, Word List Recall, 
Word List Recognition, Constructional Praxis, and Construc- 
tional Recall) included in the CERAD neuropsychological bat- 
tery were applied by experienced clinical psychologists or 
nurses. The VF-supplemented MMSE (MMSE+VF) score was 
derived from simply summing the scores of the two tests. 

A panel consisting of four psychiatrists with expertise in de- 
mentia research made the clinical decisions, including diagno- 
sis and CDR, after reviewing all the available raw data. 
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Statistical analysis 

Between-group comparisons for continuous data, including 
demographic and clinical data, were performed by two-tailed 
t tests. Categorical data were analyzed by the X 2 test. The re- 
ceiver operating characteristic (ROC) curve analysis was used 



to compare the screening accuracy for MCI, dementia and 
their subtypes between MMSE and MMSE+VF. The area un- 
der the curve (AUC) of the ROC curve was compared by us- 
ing the method of Hanley and McNeil. 21 
The level of statistical significance was set as two -tailed 



Table 1. Demographic and clinical characteristics of subjects (N=1,515) 




CN 
N=655 


MCI 
N=366 


D 
N=494 


MCI+D 
N=860 




p-value 






CN vs. MCI 


CN vs. D 


CN vs. MCI+D 




N=l,021 


N=l,149 


N=l,515 


Age (years)* 


69.5±7.5 


71.1+6.8 


72.0±8.0 


71.6+7.6 


<0.001 


<0.001 


<0.001 


Education (years)* 


7.9±5.2 


8.3±5.0 


8.215.3 


8.3+5.2 


0.283 


0.317 


0.216 


Women (%)+ 


65.0 


72.1 


62.3 


66.5 


0.020 


0.353 


0.584 


CDR(%)+ 










<0.001 


<0.001 


<0.001 


CDR0(%) 


100.0 


0.0 


0.0 


0.0 








CDR0.5(%) 


0.0 


100.0 


39.7 


65.1 








CDR 1 (%) 


0.0 


0.0 


47.4 


27.3 








CDR2+ (%) 


0.0 


0.0 


13.0 


7.4 








MMSE* 


25.3±3.2 


23.0±4.0 


16.315.7 


19.2+6.0 


<0.001 


<0.001 


<0.001 


VF* 


14.1 ±4.0 


11.4±3.8 


7.514.0 


9.214.3 


<0.001 


<0.001 


<0.001 



Data are presented as mean±SD or number (%). *by Student t-test, df=l,019; 1,147; 1,513, +by X 2 test, df = 1; 3; 3. CN: cognitively normal, 
MCI: mild cognitive impairment, D: dementia, MCI+D: overall cognitive impairment including MCI and dementia, CDR: clinical dementia 
rating, MMSE: mini-mental state examination, VF: verbal fluency using animal category 

Table 2. Area under the curves (AUC) and cutoff scores of MMSE, VF, and MMSE+VF in CN, dementia and overall cognitive impairment 
(MCl+dementia) groups 



CNvs. MCI (N= 1,021) 



CN vs. dementia (N=l,149) 



CN vs. MCI+dementia (N=l,515) 



MMSE 
AUC 
SE 

95% CI 
Cutoff 
Sen/Spe 
VF 
AUC 
SE 

95% CI 
Cutoff 
Sen/Spe 
MMSE+VF 
AUC 
SE 

95% CI 
Cutoff 
Sen/Spe 



0.68 

0.02 
0.65-0.71 

25/26 
73.5/55.4 

0.69 

0.02 
0.66-0.72 

12/13 
64.8/60.2 

0.72*t 

0.02 
0.69-0.75 

37/38 
69.4/64.6 



0.92t 

0.01 
0.90-0.93 

21/22 
79.8/88.9 

0.88 

0.01 
0.86-0.90 

10/11 
78.3/81.7 

0.93*+ 

0.01 
0.92-0.95 

33/34 
87.7/84.1 



0.82 

0.01 
0.80-0.84 

23/24 
73.6/72.7 

0.80 

0.01 
0.78-0.82 

11/12 
72.0/72.2 

0.84*+ 

0.01 
0.82-0.86 

35/36 
77.3/75.1 



*significantly greater than that of MMSE, +significantly greater than that of VF (tested by Hanley and McNeils method). MMSE: Mini-Mental 
State Examination, VF: verbal fluency using animal category, MMSE+VF: Sum of MMSE and VF, CN: cognitively normal, MCI: mild cogni- 
tive impairment, MCI+dementia: overall cognitive impairment including MCI and dementia, AD: Alzheimer's disease, NAD: non- Alzheim- 
er s disease dementia, SE: standard error, CI: confidence interval, Sen/Spe: sensitivity/ specificity 
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Figure 1. Receiver operating characteristic (ROC) curves of Mini- 
Mental State Examination (MMSE), Verbal fluency (VF) and VF-sup- 
plemented MMSE (MMSE+VF) in cognitive impairment screening for 

(A) cognitively normal (CN) versus mild cognitive impairment (MCI), 

(B) CN versus dementia and (C) CN versus overall cognitive impair- 
ment (MCI plus dementia). 



p<0.05. ROC curve analyses were performed by using Med- 
Calc for Windows, version 12.1 (MedCalc Software, Mar- 
iakerke, Belgium). All other analyses, except the ROC curve 
analysis, were performed by using SPSS software, version 15.0 
(SPSS Inc, Chicago, IL). 

RESULTS 

Demographic and clinical characteristics 

The demographic and clinical characteristics of subjects are 
summarized in Table 1. 

ROC analysis 

The ROC curve was constructed for each score, as shown in 
Figures 1 and 2, and the AUC for each ROC curve was calcu- 
lated. AUC, sensitivity specificity cutoff points of MMSE, VF 
and MMSE+VF scores are shown in Table 2 and 3. The results 



of the ROC curve comparisons between the MMSE raw score 
and MMSE+VF score are as follows: 

Screening of MCI, dementia and overall cognitive 
impairment 

MCI screening accuracy of MMSE+VF was significantly 
better than that of MMSE (z=3.486, p=0.0005) (Table 2, Figure 
1A). MMSE+VF showed a significantly superior dementia 
screening accuracy to MMSE (z=2.881, p=0.0040) (Table 2, 
Figure IB). MMSE+VF also showed a significantly superior 
overall cognitive impairment screening accuracy to MMSE 
(z=4.046, p=0.0001) (Table 2, Figure 1C). 

Screening the subtypes of MCI and dementia 

MMSE+VF showed a significantly better screening accura- 
cy for both aMCI and naMCI compared to the MMSE alone 
(z=2.992, p=0.0028 for aMCI; z=2.387, p=0.0170 for naMCI) 
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Table 3. Area under the curves (AUC) and cutoff scores of MMSE, VF and MMSE+VF in CN and subtypes of MCI and dementia 



CNvs. MCI (N=937) 



CN vs. naMCI (N=739) CN vs. AD (N= 1 ,00 1 ) 



CNvs. NAD (N=803) 



MMSE 
AUC 
SE 

95% CI 
Cutoff 
Sen/Spe 
VF 
AUC 
SE 

95% CI 
Cutoff 
Sen/Spe 
MMSE+VF 
AUC 
SE 

95% CI 
Cutoff 
Sen/Spe 



0.70 

0.02 
0.67-0.73 

25/26 
77.0/55.4 

0.70 

0.02 
0.67-0.73 

12/13 
65.3/60.2 

0.74*t 

0.02 
0.71-0.77 

37/38 
72.0/64.6 



0.59 

0.03 
0.56-0.63 

25/26 
61.9/55.4 

0.67* 

0.03 
0.63-0.70 

12/13 
63.1/60.2 

0.65* 

0.03 
0.61-0.68 

37/38 
60.7/64.58 



0.93+ 

0.01 
0.91-0.94 

21/22 
82.4/88.9 

0.87 

0.01 
0.85-0.89 

10/11 
76.6/81.7 

0.93+ 

0.01 
0.92-0.95 

33/34 
87.3/84.1 



0.89 

0.02 
0.87-0.91 

21/22 
73.7/88.9 

0.91* 

0.01 
0.89-0.93 

10/11 
82.4/81.7 

0.93*+ 

0.01 
0.91-0.95 

33/34 
88.5/84.1 



*significantly greater than that of MMSE, +significantly greater than that of VF (tested by Hanley and McNeils method). MMSE: Mini-Mental 
State Examination, VF: verbal fluency using animal category, MMSE+VF: Sum of MMSE and VF, CN: cognitively normal, MCI: mild cogni- 
tive impairment, AD: Alzheimer's disease, NAD: non- Alzheimer s disease dementia, SE: standard error, CI: confidence interval, Sen/Spe: sen- 
sitivity/ specificity 



(Table 3, Figure 2A and B). Among the 84 subjects with naM- 
CI, 9 subjects with a single domain naMCI scored worse on 
only the VF test. In order to minimize the concern, we per- 
formed additional analyses on all subjects, excluding the 9 
subjects (n= 1,506). In this operational condition, we found 
that MMSE+VF had a significantly better screening ability 
for MCI (z=3.024, p=0.0025) and overall cognitive impair- 
ment (z=3.685, p=0.0002) than MMSE. Regarding the demen- 
tia subtypes, MMSE+VF was significantly better than MMSE 
for NAD screening (z=4.186, p<0.0001) (Table 3, Figure 2D), 
but not for AD screening (z=0.810, p=0.4182) (Table 3, Figure 
2C). 

DISCUSSION 

We found that MMSE+VF had a significantly better screen- 
ing ability for MCI, dementia and overall cognitive impair- 
ment compared to the MMSE alone. For additional subtype 
analyses for MCI and dementia, we also found that it showed a 
significantly better ability than MMSE for both MCI subtypes 
and NAD screening, but not for AD screening. 

MMSE also has poor sensitivity in detecting MCI, and is 
less sensitive to frontal executive dysfunction. Standish, Molloy, 
Cunje and Lewis 22 found that a subtest of fluency (type not 



specified) was best able to discriminate between CN and MCI 
subjects after controlling for age and education. Saxton et al 23 
suggested that semantic fluency (fruit category) showed a sig- 
nificant baseline decline in 1.5-5 years prior to the onset of 
AD. Furthermore, sematic fluency involves various executive 
functions which support or enable the semantic retrieval pro- 
cesses, such as systematic searching and avoiding persevera- 
tions and non-category intrusions. 24,25 These imply that the 
supplementation of VF has been proposed to improve the 
MCI and dementia screening accuracy of MMSE. 

In the MCI screening, we performed an ROC curve analysis 
that showed the AUC of 0.718 for MMSE+VF in spite of the 
VF-supplementation, which improves the screening ability of 
MMSE. This finding may be explained by two possibilities. 
First, this modest level of AUC is influenced by poor sensitivi- 
ty in detecting MCI by MMSE itself rather than by the VF- 
supplementation effect. Second, the MCI group is comprised 
of subjects with very heterogeneous cognitive impairments 
rather than homogeneously prominent executive deficits. 

There is several evidence that executive deficits are an im- 
portant feature of the cognitive decline as well as episodic 
memory deficits in AD, and that these deficits typically occur 
early in the disease, and may be the first non-memory deficits 
to occur. 26,27 However, in our study, we discovered the VF-sup- 
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Figure 2. Receiver operating characteristic (ROC) curves of Mini-Mental State Examination (MMSE), Verbal Fluency (VF) and VF-supple- 
mented MMSE (MMSE+VF) in cognitive impairment screening for (A) cognitively normal (CN) versus amnestic mild cognitive impairment 
(aMCI), (B) CN versus non-amnestic MCI (naMCI), (C) CN versus Alzheimer's disease (AD) and (D) CN versus non-Alzheimer's disease 
dementia (NAD). 



plementation effect only for NAD screening, not for AD 
screening. This is probably associated with the fact that the 
frontal executive function is relatively more impaired in NAD, 
particularly FTD 28,29 and VD8 compared to AD. Furthermore, 
the lowered VF-supplementation effect on AD screening in 
this study is likely to be a reflection of the heterogeneous na- 
ture of AD that may include subjects with subtle executive def- 
icits, which may be undetected by VF-supplementation perfor- 
mance. 

Several scales have been proposed to overcome the short- 
comings of MMSE in MCI and NAD screening by inserting or 
modified VF, such as the Concise Cognitive Test (CON- 
COG), 30 the 7-Minutes Screen (7MS) 31 and the Montreal Cog- 
nitive Assessment (MoCA). 32 Srinivasan30 suggested that 



CONCOG has negligible ceiling effects compared with the 
MMSE. Meulen et al. 33 demonstrated that the sensitivity of the 
7MS for NAD was 89.4% with a specificity of 93.5%. Dong et 
al. 34 indicated that MoCA is superior to the MMSE in the 
screening of patients with MCI who are at a higher risk for in- 
cident dementia. Lee et al. 35 suggested that MoCA had an ex- 
cellent sensitivity of 89% and good specificity of 84% for MCI 
screening. Though these scales have demonstrated better 
screening accuracy for MCI or NAD compared with the 
MMSE, these scales are not widely used in Korean clinical set- 
tings due to their unfamiliar westernized items. Therefore, VF- 
supplementation was selected to improve the screening ability 
of MMSE for MCI and NAD in our study. 
The strengths of this study lead us to believe that our find- 
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ings will be replicated in other settings and populations. First, 
our study population was rather large and had strict diagnoses 
of CN, MCI, dementia and their subtypes. These were con- 
ducted through a clinical evaluation using the strict diagnostic 
criteria by a panel consisting of four psychiatrists [two panel 
members (DYL and KWK) were certified as CDR raters by 
the Memory and Aging Project of the AD Research Center at 
the Washington University School of Medicine] with expertise 
in this area. This may have increased the reliability and gener- 
alizability of our data. Furthermore, MMSE+VF has an im- 
portant cost-effect perspective related with the superior MCI 
and NAD screening accuracy but with a little burden on clini- 
cians. Moreover, it could be broadly applied in some special- 
ized clinical settings (e.g., stroke clinic) with benefits due to its 
superior NAD screening accuracy. 

In conclusion, our findings strongly support the usefulness 
of VF-supplementation to improve the screening performance 
of MMSE for MCI and NAD. 
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